Analyzing large-scale data processing jobs

ABSTRACT

Methods, systems, and apparatus for data analysis in a distributed computing system by accessing data stored at a first processing zone associated with a distributed data processing job, detecting information identifying a particular child job associated with the distributed data processing job, comparing the identifying information to data stored at a second processing zone, and identifying an additional child job as associated with the distributed data processing job based on a result of the comparison. The methods, systems and apparatus are further for correlating particular output data associated with the particular child job and additional output data associated with the additional child job for the distributed data processing job, determining performance data for the distributed data processing job based on the output data associated with each of the particular child job and the additional child job, and providing for display the performance data for the distributed data processing job.

CROSS REFERENCE TO RELATED APPLICATIONS

This U.S. patent application is a continuation of, and claims priorityunder 35 U.S.C. § 120 from, U.S. patent application Ser. No. 17/095,718,filed on Nov. 11, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/708,897, now U.S. Pat. No. 10,860,454, filed onDec. 10, 2019, which is a continuation of U.S. patent application Ser.No. 15/432,375, now U.S. Pat. No. 10,514,993, filed on Feb. 14, 2017.The disclosures of these prior art applications are considered part ofthe disclosure of this application and hereby incorporated by referencein their entireties.

TECHNICAL FIELD

This disclosure relates to analyzing large scale data processing jobs.

BACKGROUND

Large scale data processing has become widespread in web companies andacross industries. Large-scale data processing may include parallelprocessing, which generally involves performing some operation over eachelement of a large data set simultaneously. The various operations maybe chained together in a data-parallel pipeline to create an efficientmechanism for processing a data set. Production of the data set mayinvolve creation of child jobs or stages that execute for the main orparent job, where each child job may execute on different processingzones. Given the size of the large scale data processing jobs, however,it is difficult to analyze the performance of the large scale jobs.

SUMMARY

The present specification generally relates to large-scale dataprocessing jobs.

Diagnosing anomalies in data processing pipelines may be difficult toachieve after the pipeline finishes running. Some of the challengesinclude missing logs, difficulty collating data across multiple runs,correlating information with other processing events, and determining arelationship between a main job and the stages or child jobs of thatmain job. There may be an array of relevant diagnostic information andanalysis including pipelines failures, slowness, and performancemetrics. Thus, there is a need for a data analyzing tool that enablescollection of relevant information regarding a distributed dataprocessing job and enables diagnosis of anomalies in data pipelines.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof accessing data, stored in a storage device of a first processingzone, that is associated with a particular distributed data processingjob that has been executed; detecting, from the data stored in thestorage device, identifying information that identifies a particularchild job associated with the particular distributed data processingjob; in response to detecting the identifying information thatidentifies a particular child job associated with the particulardistributed data processing job, comparing the identifying informationto data stored in a storage device of a second processing zone;identifying an additional child job as being associated with theparticular distributed data processing job based on a result ofcomparing the identifying information to data stored in the storagedevice of the second processing zone; correlating particular output dataassociated with the particular child job and additional output dataassociated with the additional child job for the particular distributeddata processing job; determining performance data for the particulardistributed data processing job based on the particular output dataassociated with the particular child job and the additional output dataassociated with the additional child job; and providing for display theperformance data for the particular distributed data processing jobbased on the particular output data associated with the particular childjob and the additional output data associated with the additional childjob.

In certain implementations, the methods further include the actions ofcomparing performance data for the particular distributed dataprocessing job to a performance threshold; and providing a notificationbased on a result of comparing performance data for the particulardistributed data processing job to the performance threshold.

In certain implementations, the notification includes one or more of: anaudible alert, a tactile alert, a visual alert, or an electronicmessage. In certain implementations, the performance data includes oneor more of: a running time, memory usage, CPU time, disk usage, arelationship between each child job and the particular distributed dataprocessing job, one or more counters associated with the particulardistributed data processing job, or a processing status.

In certain implementations, the methods further include the actions ofdisplaying a user interface that includes display of the performancedata, wherein the user interface includes an interactive hierarchicalstructure.

In certain implementations, the identifying information includes acommon prefix identified in the data.

In certain implementations, the particular distributed data processingjob is associated with a particular pipeline; and correlating particularoutput data associated with the particular child job and additionaloutput data associated with the additional child job for the particulardistributed data processing job includes associating the particularchild job and the additional child job with the particular pipeline; andthe methods include the actions of: determining pipeline performancedata for a first run of the particular pipeline; and determiningpipeline performance data for a second run of the particular pipeline.

Other realizations of the above aspects include corresponding systems,apparatus, and computer programs, configured to perform the actions ofthe methods, encoded on computer storage devices.

Particular implementations of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. The systems and methods allow for a mechanism toreliably and accurately analyze the correctness and performance oflarge-scale data processing jobs. The systems and methods realized anintuitive data analysis tool that enables the diagnosing of pipelineflow errors and pipeline accuracy more quickly and accurately than insystems that do not implement these systems and methods. This allows forremedial actions to be more focused and efficient, which saves both timeand system resources. Further, implementations of the present disclosureachieve technical advantages such as identifying child job data relatedto a main job when the child job data is stored across processing zones,automated collection and correlation of child job data stored acrossprocessing zones, more efficient and faster processing for thecollection and correlation of performance data, automated notificationsor alerts regarding job performance metrics, and analysis and diagnosisof job performance including an intuitive user interface. In particular,implementations of the present disclosure achieve the technicaladvantage of more easily identifying root causes of performancedegradation and identifying stages that use more resources overdifferent runs.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otherfeatures and advantages of the disclosure will become apparent from thedescription, the drawings, and the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 depicts an example of a distributed data processing system,according to implementations of the present disclosure.

FIG. 2 depicts an example of a system for analyzing data of adistributed data processing system, according to implementations of thepresent disclosure.

FIG. 3 depicts an example of a processing user interface, according toimplementations of the present disclosure.

FIG. 4 depicts a flowchart of an example of a process for analyzing datain a distributed data processing system, according to implementations ofthe present disclosure.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

At a high level, implementations of the present disclosure provide anoffline pipeline analysis and diagnosis framework, which collects andorganizes relevant data related to distributed data processing pipelinesstored across different processing zones. In addition, the data iscorrelated and presented in a more structured form to the user and mayprovide automated notification regarding certain performance metrics.Some example performance analysis includes, but is not limited to: thenumber of times the pipeline ran in the last n amount of time and thecorresponding status, run times for different stages or different phasesof the data processing job, variations in counters across different runsof the pipeline, the number of unique failures encountered for a givenpipeline and any existing bugs, a change in processing environment forthe pipeline such as grouping, scheduling, strategy across differentruns of the pipeline, and other transient issues that may affect thepipeline. One example of a distributed data processing job is aMapReduce job that may include map, shuffle, and reduce phases. However,other distributed processing systems may also benefit fromimplementations of the present disclosure.

These features and additional features are described in more detailbelow.

FIG. 1 depicts an example of a distributed data processing system 100,according to implementations of the present disclosure. The distributeddata processing system 100 may include a data processor 102, which mayinclude one or more computers. The data processor 102 may store data,for example, across storage nodes 110 at processing zones 104, 106, and108. Conventional processing zones can store large amounts of data. Somedata is stored redundantly across multiple processing zones so that evenif an entire processing zone fails the data can be recovered. The dataprocessor 102 may communicate with the processing zones 104, 106, and108 using a network 112.

A storage node may include one or more computer storage mediums. In someimplementations, a storage node is a data server, for example, a serverincluding a data processing apparatus and multiple hard disk drives onwhich data can be stored. A group of storage nodes may include a rack, asubnetwork, a processing zone, or various other collections of serversor storage nodes.

A processing zone may include a grouping of storage nodes. A processingzone is designed to be relatively independent from other processingzones. For example, a processing zone may have independent resources,such as power, networking, environmental controls, security, or thelike, or any combination thereof. Processing zones may include, forexample, networking systems, backup power supplies, climate controls,security, or the like, or any combination thereof. A processing zone mayinclude or be limited to a single facility or building or one or morefacilities, or in some instances may include or be limited to a portionof a single facility. In FIG. 1 , the processing zones 104, 106, and 108are shown with three storage nodes; however, each processing zone canhave more or fewer storage nodes. Data may be stored in data chunks,each data chunk including a specified amount of data. In someimplementations, a data chunk may be a contiguous portion of data from afile. In some other implementations, a data chunk may be one or morenon-contiguous portions of data from a file. In some implementations,metadata is used at the processing zones 104, 106, and 108 to keep trackof stored data. For example, the metadata may specify which parts of afile are stored at which processing zones. In addition, data chunks maybe specified to be stored at certain processing zones. For example, datamay be specified to be stored at a processing zone based on theprocessing zone's geographic location.

In a distributed data processing system, for example distributed dataprocessing system 100 of FIG. 1 , a particular distributed dataprocessing job may include a main job that creates multiple child jobs,which may span multiple processing zones 104, 106, and 108. These jobsmay be referred to as batch jobs. Because the child jobs span multipleprocessing zones, accessing the data at different locations is difficultwith conventional systems and processes. In particular, it may not bepossible to obtain consolidated data for the entire job to provideaccurate performance analysis and diagnostics with these conventionalsystems and processes. In particular, for example, the metadatadescribed above may be lost or deleted once a distributed dataprocessing job is completed. Thus, a correlation between various childjobs and the main job may be difficult to ascertain.

For example, in a typical MapReduce system, the MapReduce framework willautomatically split the job into multiple child jobs, sometimes executedin parallel. The relationships between the main job and child jobs arenot readily attainable, especially for a job that has terminated.Further, for example, it is difficult to track and correlate countersregarding the number of objects and operations the MapReduce job isprocessing, which serve as indicators of the job's behavior and to helpwith debugging the job, as well as information about the code versionused to build the MapReduce job binary. In addition, for example, aMapReduce job as part of a large scale data processing MapReduce systemis typically run on a regular basis, but it is difficult to view therelative performance of the same job over time, for example when changesare introduced to the MapReduce system as part of the developmentprocess. Moreover, it is difficult to obtain information regarding theMapReduce job's interaction with the distributed runtime environment.However, implementations of the present disclosure enable each of thesedifficulties or challenges to be overcome, thereby providing theassociated functionality for the performance and diagnosis of alarge-scale data processing system, as described in more detail below.

Implementations of the present disclosure provide a manner forcollecting and correlating relevant information regarding each of thechild jobs, determining a relationship between respective child jobs andthe main job, determining how different jobs are executing relative toeach other, and enabling developers to identify which stage or phase isexecuted in each job, even when the data is stored across differentprocessing zones. In addition, implementations of the present disclosuremay collect and group data relevant to different runs of the samepipeline to provide enhanced diagnostic and analysis capabilities. Forexample, as described in more detail below, a footprint or pattern maybe determined from data associated with a distributed data processingjob that may be used to identify child jobs stored across differentprocessing zones as being associated with the main job. For example, thefootprint or pattern may include a common prefix that uniquelyidentifies child jobs associated with the main job. In this manner, dataassociated with child jobs may be identified and correlated with eachother and with the main job, and from that data, performance informationmay be determined.

FIG. 2 depicts an example of a system 200 for analyzing data of adistributed data processing system, according to implementations of thepresent disclosure. As an example of distributed data processing, aprocessing pipeline 210 is depicted in FIG. 2 . The pipeline 210 maycommunicate with and store data in one or both of a distributed dataprocessing job information database 220 and a pipeline informationdatabase 230. As described above, these databases may be stored instorage devices across processing zones. For example, a portion of dataassociated with pipeline 210 may be stored at one processing zone andanother portion of data associated with pipeline 210 may be stored atanother processing zone. Further, for example, each of those portions ofdata may be associated with a respective child job of the pipeline 210.

Data from one or both of the distributed data processing job informationdatabase 220 and the pipeline information database 230 may be accessedby data processor 240. One of the main functions for data processor 240is a data collection and processing pipeline 250 which collects andprocesses all the pipeline relevant information and writes that data toprocessing database 260. The data collection and processing pipeline 250may collect information from a number of sources, including thedistributed data processing job information database 220 and thepipeline information database 230, which may be accessed by dataprocessor 240. In addition, the data collection and processing pipeline250 may collect information from external services and the processingenvironment 280. The processing user interface (UI) 270 providesinformation for display from the data collected and processed by thedata collection and processing pipeline 250 that is stored in theprocessing database 260.

The processing pipeline 250 may collect information related to each jobfrom multiple sources, including log files, log databases, event logs,and runtime environment settings, which may then be presented on theprocessing UI 270. For example, the processing pipeline 250 may collectinformation at a predetermined time interval. The information identifiedand collected by processing pipeline 250 may be stored in processingdatabase 260. That data may also be optimized for combining with otherdata before or after being stored to enable more efficient processing ofthe data for presentation on the processing UI 270.

Some examples of the type of information that processing pipeline 250may collect include, but are not limited to: the running time of eachjob; the memory usage, CPU time, and disk usage for each run of a job;information regarding each stage or phase of the job for each run of thejob; time elapsed for each stage or phase, along with memory usage, CPUtime, and disk usage of each stage or phase; a “parent-child”relationship between the main job and the child jobs or stages, whichmay be collected and reconstructed from the log files and log databasesinto which the system writes logging information; any counterinformation logged in each stage or phase, which may be collected fromlog files or databases, or from a different database where the counterinformation may be stored due to its large volume.

The processing UI 270, for display of the performance informationdetermined by processing pipeline 250, may be structured in a manner toenable a user to selectively view different levels of detail for theinformation associated with the job. For example, the processing UI 270may include a hierarchy of information displayed so that a user mayselectively view higher level information about the system or job, andmake a selection to view more detailed information about the system orjob or about a portion of the system or job. This interactive hierarchyof information to be displayed is described in more detail withreference to FIG. 3 , showing an example of a processing UI, which hasbeen described above.

FIG. 3 depicts an example of a processing UI 300, according toimplementations of the present disclosure. For example, the processingUI 300 may include a dashboard UI 305. The dashboard UI 305 may includean interactive hierarchy of levels of information that allow a user toselectively drill down into the job's detailed information. Thisprocessing UI 300 enables a user to more easily observe the status ofall of the runs of a particular job, compare different runs, comparedifferent jobs, or the like, or any combination thereof.

For example, the first level UI 310 may display all the distinct jobsassociated with a user or that the user has selected to be displayed.The first level UI 310 may also display the overall number of jobs thathave passed and failed in a particular timeframe. Further, for example,the first level UI may display a number of executions by state and atotal duration by workflow. As part of the processing UI 300, a user mayselect each distinct job, which may provide a display of the next levelof information, the second level UI 320.

The second level UI 320 may display historical data regarding elapsedtime and resources consumed by runs of a particular job over aparticular timeframe. The second level UI 320 may also display anynumber of the following performance metrics: pipeline run duration,pipeline CPU usage time, pipeline memory usage, pipeline disk usage,instances or number of pipeline runs, stage duration by run, a list ofthe n longest average stage durations, and a list of stage duration byname. Further, for example, as a list of the instances of past job runsmay be presented on the second level UI 320, a user may select any ofthose instances, which may provide a display of the next level ofinformation, the third level UI 330.

The third level UI 330 may display information regarding one specificrun instance of the job. For example, the third level UI 330 may displayinformation regarding running time, CPU usage time, memory usage, anddisk usage of each stage within the job as well as the binary buildversion for the job. The third level UI 330 may also display pipelinestates, a pipeline start time and duration, pipeline stages, and countervalues associated with the instance of the job. In addition, a timelineof each stage's start and stop may be displayed on the third level UI330.

As another feature of the second level UI 320, a user may select twodifferent runs from those displayed and select to compare these runswith each other. For example, that selection may cause a comparison UIto be displayed, in which information regarding the two selected runs isdisplayed side-by-side for easier comparison. The comparison UI may bepart of the second level UI 320 or may be a separate UI displayed withinprocessing UI 300.

FIG. 4 depicts a flowchart of an example of a process 400 for dataanalysis in a distributed data processing system, according toimplementations of the present disclosure. Although process 400 isdepicted and described with steps occurring in a particular order, oneor more of those steps may occur in a different order than what isdepicted and described.

Once a large-scale data processing job is completed, information relatedto the job may be stored across different processing zones and difficultto identify, extract, and analyze. According to implementations of thepresent disclosure, data stored in one or more storage devices at afirst processing zone may be accessed at 410. The data accessed at 410may be data associated with a particular distributed data processing jobthat has been executed.

At 420, identifying information that identifies a particular child jobassociated with the particular distributed data processing job may bedetected from data stored in a storage device at the first processingzone. The identifying information may include a pattern that uniquelyidentifies a child job associated with the particular distributed dataprocessing job. For example, the pattern may be a common prefixidentified in the data that may be common to other child jobs associatedwith the particular distributed data processing job.

In response to detecting the identifying information that identifies aparticular child job associated with the particular distributed dataprocessing job, the identifying information may be compared to datastored in a storage device of a second processing zone at 430. Like thefirst processing zone, the second processing zone may include one ormore storage devices on which data regarding one or more child jobs isstored. At 440, an additional child job may be identified as beingassociated with the particular distributed data processing job based ona result of the comparison between the identifying information thatidentifies a particular child job associated with the particulardistributed data processing job and the data stored in the storagedevice of the second processing zone.

At 450, particular output data associated with the particular child jobmay be correlated with additional output data associated with theadditional child job for the particular distributed data processing job.For example, one or more relationships between child jobs and respectiverelationships between each child job and the main job may be determined.

Further, at 460, performance data for the particular distributed dataprocessing job may be determined based on the particular output dataassociated with the particular child job and the additional output dataassociated with the additional child job. The performance data mayinclude: a running time, memory usage, CPU time, disk usage, arelationship between each child job and the particular distributed dataprocessing job, one or more counters associated with the particulardistributed data processing job, a processing status, or the like, orany combination thereof.

Once the performance data for the particular distributed data processingjob is determined, that performance data may be provided for display, at470, based on the particular output data associated with the particularchild job and the additional output data associated with the additionalchild job. For example, various performance information and metrics maybe calculated from the collected data and may be displayed on aninteractive UI. Further, for example, performance data for theparticular distributed data processing job may be compared to aperformance threshold, and a notification may be provided based on aresult of the comparison of the performance data for the particulardistributed data processing job to the performance threshold. Forexample, the notification may include an audible alert, a tactile alert,a visual alert, an electronic message, or the like, or any combinationthereof.

As noted above, the performance data provided for display at 270 may beprovided via an UI that includes display of the performance data. The UImay include an interactive hierarchical structure. In that manner, theUI may selectively display varying levels of detail or informationregarding different jobs, different runs of a pipeline, different childjobs, or the like, based on a user's selection. For example, asdescribed above with reference to FIG. 3 , the UI may include differenthierarchical levels of display with which a user may interact and whichmay be selectively displayed based on the type of information the userdesires.

In implementations of the present disclosure, the particular distributeddata processing job may be associated with a particular distributed dataprocessing pipeline. Thus, for example, correlating particular outputdata associated with the particular child job and additional output dataassociated with the additional child job for the particular distributeddata processing job may include associating the particular child job andthe additional child job with the particular pipeline. In addition, forexample, pipeline performance data may be determined for a first run ofthe particular pipeline, and pipeline performance data may be determinedfor a second run of the particular pipeline.

Thus, a comparison may be made between different runs of a particularpipeline, and the performance data may be provided on the UI such thatthe different runs of the particular pipeline may be comparedside-by-side. Accordingly, a user may more easily identifydiscrepancies, latencies, differences in resource consumption, or thelike, or any combination thereof. Identifying the relationship betweendifferent runs of the same pipeline enables the comparison to be made,which provides a more efficient diagnosis of issues for the distributeddata processing pipeline.

Thus, implementations of the present disclosure achieve technicaladvantages such as identifying child job data related to a main job whenthe child job data is stored across processing zones, automatedcollection and correlation of child job data stored across processingzones, more efficient and faster processing for the collection andcorrelation of performance data, automated notifications or alertsregarding job performance metrics, and analysis and diagnosis of jobperformance including an intuitive user interface. Further, certainadvantages and technical effects of implementations of the presentdisclosure include an interactive UI for a user to observe the status ofall job runs, the ability to compare two job runs side-by-side to moreeasily investigate possible root causes of any performance degradationin the job, automatically identifying a job that decreases inperformance over time, and identifying stages within a job that aretaking the longest time to run or consuming the most computingresources. Thus, deficiencies or problems in a distributed dataprocessing job may be more easily and quickly diagnosed and corrected,thereby decreasing processing time and improving performance of thedistributed data processing system.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, various formsof the flows shown above may be used, with steps re-ordered, added, orremoved.

Implementations of the disclosure and all of the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Implementationsof the disclosure can be implemented as one or more computer programproducts, i.e., one or more modules of computer program instructionsencoded on a computer readable medium for execution by, or to controlthe operation of, data processing apparatus. The computer readablemedium can be a machine-readable storage device, a machine-readablestorage substrate, a memory device, a composition of matter effecting amachine-readable propagated signal, or a combination of one or more ofthem.

The term “data processing apparatus” encompasses all apparatus, devices,and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them. A propagated signal is an artificially generated signal, e.g.,a machine-generated electrical, optical, or electromagnetic signal thatis generated to encode information for transmission to suitable receiverapparatus.

While this disclosure contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular implementations of the disclosure. Certain features that aredescribed in this specification in the context of separateimplementations can also be implemented in combination in a singleimplementation. Conversely, various features that are described in thecontext of a single implementation can also be implemented in multipleimplementations separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular implementations of the present disclosure have beendescribed. Other implementations are within the scope of the followingclaims. For example, the actions recited in the claims can be performedin a different order and still achieve desirable results. A number ofimplementations have been described. Nevertheless, it will be understoodthat various modifications may be made without departing from the spiritand scope of the disclosure. For example, various forms of the flowsshown above may be used, with steps re-ordered, added, or removed.Accordingly, other implementations are within the scope of the followingclaims.

What is claimed is:
 1. A computer-implemented method when executed bydata processing hardware causes the data processing hardware to performoperations comprising: obtaining job data associated with a distributeddata processing job that has been executed at a distributed dataprocessing environment comprising a first processing zone and a secondprocessing zone; identifying, based on the job data, a first child joband a second child job each created by the distributed data processingjob, the first child job associated with first output data and the firstprocessing zone, the second child job associated with second output dataand the second processing zone; determining, based on the first outputdata and the second output data, performance data of the distributeddata processing job; determining that the performance data satisfies aperformance threshold; and based on determining that the performancedata satisfies the performance threshold, performing an action.
 2. Thecomputer-implemented method of claim 1, wherein the operations furthercomprise: receiving, from a user of the distributed data processingenvironment, a user selection selecting the performance data of thedistributed data processing job; and in response to receiving the userselection selecting the performance data of the distributed dataprocessing job, displaying the performance data at a display incommunication with the data processing hardware.
 3. Thecomputer-implemented method of claim 2, wherein displaying theperformance data comprises displaying the performance data in aninteractive hierarchical structure.
 4. The computer-implemented methodof claim 1, wherein the operations further comprise correlating thefirst output data and the second output data based on the job data. 5.The computer-implemented method of claim 1, wherein performing theaction comprises generating a notification.
 6. The computer-implementedmethod of claim 1, wherein the job data comprises first job data storedat a first storage device at the first processing zone.
 7. Thecomputer-implemented method of claim 1, wherein the job data comprisessecond job data stored at a second storage device at the secondprocessing zone.
 8. The computer-implemented method of claim 1, whereinthe performance data comprises a counter associated with the distributeddata processing job.
 9. The computer-implemented method of claim 1,wherein the performance data comprises a change of processingenvironment of the distributed data processing environment.
 10. Thecomputer-implemented method of claim 1, wherein the first processingzone and the second processing zone each comprise a respective groupingof storage nodes.
 11. A system comprising: data processing hardware of adistributed computing system; and memory hardware in communication withthe data processing hardware, the memory hardware storing instructionsthat when executed on the data processing hardware cause the dataprocessing hardware to perform operations comprising: obtaining job dataassociated with a distributed data processing job that has been executedat a distributed data processing environment comprising a firstprocessing zone and a second processing zone; identifying, based on thejob data, a first child job and a second child job each created by thedistributed data processing job, the first child job associated withfirst output data and the first processing zone, the second child jobassociated with second output data and the second processing zone;determining, based on the first output data and the second output data,performance data of the distributed data processing job; determiningthat the performance data satisfies a performance threshold; and basedon determining that the performance data satisfies the performancethreshold, performing an action.
 12. The system of claim 11, wherein theoperations further comprise: receiving, from a user of the distributeddata processing environment, a user selection selecting the performancedata of the distributed data processing job; and in response toreceiving the user selection selecting the performance data of thedistributed data processing job, displaying the performance data at adisplay in communication with the data processing hardware.
 13. Thesystem of claim 12, wherein displaying the performance data comprisesdisplaying the performance data in an interactive hierarchicalstructure.
 14. The system of claim 11, wherein the operations furthercomprise correlating the first output data and the second output databased on the job data.
 15. The system of claim 11, wherein performingthe action comprises generating a notification.
 16. The system of claim11, wherein the job data comprises first job data stored at a firststorage device at the first processing zone.
 17. The system of claim 11,wherein the job data comprises second job data stored at a secondstorage device at the second processing zone.
 18. The system of claim11, wherein the performance data comprises a counter associated with thedistributed data processing job.
 19. The system of claim 11, wherein theperformance data comprises a change of processing environment of thedistributed data processing environment.
 20. The system of claim 11,wherein the first processing zone and the second processing zone eachcomprise a respective grouping of storage nodes.